---
id: running-tracetest-with-grafana-tempo-pokeshop
title: Pokeshop and Grafana Tempo
description: Quick start on how to configure the Pokeshop Demo app with OpenTelemetry traces, Grafana Tempo as a trace data store, and Tracetest for enhancing your E2E and integration tests with trace-based testing.
hide_table_of_contents: false
keywords:
  - tracetest
  - trace-based testing
  - observability
  - distributed tracing
  - testing
  - grafana tempo
  - grafana traces
  - grafana observability
  - opentelemetry
image: https://res.cloudinary.com/djwdcmwdz/image/upload/v1698686403/docs/Blog_Thumbnail_14_rsvkmo.jpg
---

:::note
[Check out the source code on GitHub here.](https://github.com/kubeshop/tracetest/tree/main/examples/tracetest-grafana-tempo-pokeshop)
:::

[Tracetest](https://tracetest.io/) is a testing tool based on [OpenTelemetry](https://opentelemetry.io/) that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.

[Grafana Tempo](https://grafana.com/oss/tempo/) is an open-source, high-scale distributed tracing data store. It utilizes object storage to minimize costs, allowing you to store more information at a lower cost. It is deeply integrated with Grafana, Prometheus, and Loki. Grafana Tempo is also available as a paid hosted service at Grafana Cloud.

[Pokeshop API](https://docs.tracetest.io/live-examples/pokeshop/overview) is a testing ground, the team at Tracetest has implemented a sample instrumented API around the [PokeAPI](https://pokeapi.co/).

## Pokeshop API with Grafana Tempo, OpenTelemetry and Tracetest

This is a simple quick start guide on how to configure a fully instrumented API to be used with Tracetest for enhancing your E2E and integration tests with trace-based testing. The infrastructure will use Grafana Tempo as the trace data store and the Pokeshop API to generate the telemetry data.

## Prerequisites

**Tracetest Account**:

- Sign up to [`app.tracetest.io`](https://app.tracetest.io) or follow the [get started](/getting-started/overview) docs.
- Have access to the environment's [agent API key](https://app.tracetest.io/retrieve-token).

**Docker**: Have [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) installed on your machine.

## Run This Quckstart Example

The example below is provided as part of the Tracetest project. You can download and run the example by following these steps:

Clone the Tracetest project and go to the Grafana Tempo Node.js Quickstart:

```bash
git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/tracetest-grafana-tempo-pokeshop
```

Follow these instructions to run the quick start:

1. Copy the `.env.template` file to `.env`.
2. Fill out the [TRACETEST_TOKEN and ENVIRONMENT_ID](https://app.tracetest.io/retrieve-token) details by editing your `.env` file.
3. Run `docker compose run tracetest-run`.
4. Follow the links in the output to view the test results.

Follow the sections below for a detailed breakdown of what the example you just ran did and how it works.

## Project Structure

The project is built with Docker Compose.

### 1. Tracetest

The project contains [Tracetest Agent](/getting-started/install-agent), [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main), and a Node.js app.

The `docker-compose.yaml` file in the root directory of the quick start runs the Node.js app and the [Tracetest Agent](/concepts/agent) setup.

Configure the `.env` as shown below.

```bash
# Get the required information here: https://app.tracetest.io/retrieve-token

TRACETEST_TOKEN="<YOUR_TRACETEST_TOKEN>"
TRACETEST_ENVIRONMENT_ID="<YOUR_ENV_ID>"
```

### 2. Grafana Tempo

The `grafana.config.yaml` and `tempo.config.yaml` in the root directory contains all files required to configure Grafana and Tempo.

### Docker Compose Network

All `services` in the `docker-compose.yaml` are on the same network and will be reachable by hostname from within other services. E.g. `tracetest:4317` in the `tracetest/collector.config.yaml` will map to the `tracetest-agent` service, where the port `4317` is the port where Tracetest accepts telemetry data.

## Pokeshop API

The Pokeshop API is a fully instrumented REST API that makes use of different services to mimic a real life scenario.

It is instrumented using the [OpenTelemetry standards for Node.js](https://opentelemetry.io/docs/instrumentation/js/getting-started/nodejs/), sending the data to the ADOT collector that will be pushing the telemetry information to both the AWS X-Ray service.

This is a fragment from the main tracing file from the [Pokeshop API repo.](https://github.com/kubeshop/pokeshop)

```typescript
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import * as opentelemetry from '@opentelemetry/api';
import { api, NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { Resource } from '@opentelemetry/resources';
import * as dotenv from 'dotenv';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
import { IORedisInstrumentation } from '@opentelemetry/instrumentation-ioredis';
import { PgInstrumentation } from '@opentelemetry/instrumentation-pg';
import { AmqplibInstrumentation } from '@opentelemetry/instrumentation-amqplib';
import { SpanStatusCode } from '@opentelemetry/api';
import { B3Propagator } from '@opentelemetry/propagator-b3';

dotenv.config();
api.propagation.setGlobalPropagator(new B3Propagator());

const { COLLECTOR_ENDPOINT = '', SERVICE_NAME = 'pokeshop' } = process.env;

let globalTracer: opentelemetry.Tracer | null = null;

async function createTracer(): Promise<opentelemetry.Tracer> {
  const collectorExporter = new OTLPTraceExporter({
    url: COLLECTOR_ENDPOINT,
  });

  const spanProcessor = new BatchSpanProcessor(collectorExporter);
  const sdk = new NodeSDK({
    traceExporter: collectorExporter,
    // @ts-ignore
    instrumentations: [new IORedisInstrumentation(), new PgInstrumentation(), new AmqplibInstrumentation()],
  });

  sdk.configureTracerProvider({}, spanProcessor);
  sdk.addResource(
    new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: SERVICE_NAME,
    })
  );

  await sdk.start();
  process.on('SIGTERM', () => {
    sdk
      .shutdown()
      .then(
        () => console.log('SDK shut down successfully'),
        err => console.log('Error shutting down SDK', err)
      )
      .finally(() => process.exit(0));
  });

  const tracer = opentelemetry.trace.getTracer(SERVICE_NAME);

  globalTracer = tracer;

  return globalTracer;
}
```

The `docker-compose.yaml` file includes the definitions for all of the required services by the Pokeshop API, which includes:

- [**OpenTelemetry Collector**)](https://opentelemetry.io/docs/collector/getting-started/) - Vendor-agnostic way to receive, process and export telemetry data.
- **Postgres** - To save Pokemon information.
- **Redis** - For in memory strage.
- **RabbitMQ** - For async processing use cases.
- **API** - The Pokeshop API main container.
- **Worker** - The Pokeshop worker instance.

```yaml
version: "3"

#...

services:
  #...

  # Demo
  postgres:
    image: postgres:14
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
    healthcheck:
      test: pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"
      interval: 1s
      timeout: 5s
      retries: 60
    ports:
      - 5432:5432

  demo-cache:
    image: redis:6
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 1s
      timeout: 3s
      retries: 60

  demo-queue:
    image: rabbitmq:3.8-management
    restart: unless-stopped
    healthcheck:
      test: rabbitmq-diagnostics -q check_running
      interval: 1s
      timeout: 5s
      retries: 60

  demo-api:
    image: kubeshop/demo-pokemon-api:latest
    restart: unless-stopped
    pull_policy: always
    environment:
      REDIS_URL: demo-cache
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
      RABBITMQ_HOST: demo-queue
      POKE_API_BASE_URL: https://pokeapi.co/api/v2
      COLLECTOR_ENDPOINT: http://otel-collector:4317
      NPM_RUN_COMMAND: api
    ports:
      - "8081:8081"
    healthcheck:
      test: ["CMD", "wget", "--spider", "localhost:8081"]
      interval: 1s
      timeout: 3s
      retries: 60
    depends_on:
      postgres:
        condition: service_healthy
      demo-cache:
        condition: service_healthy
      demo-queue:
        condition: service_healthy

  demo-worker:
    image: kubeshop/demo-pokemon-api:latest
    restart: unless-stopped
    pull_policy: always
    environment:
      REDIS_URL: demo-cache
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
      RABBITMQ_HOST: demo-queue
      POKE_API_BASE_URL: https://pokeapi.co/api/v2
      COLLECTOR_ENDPOINT: http://otel-collector:4317
      NPM_RUN_COMMAND: worker
    depends_on:
      postgres:
        condition: service_healthy
      demo-cache:
        condition: service_healthy
      demo-queue:
        condition: service_healthy

  demo-rpc:
    image: kubeshop/demo-pokemon-api:latest
    restart: unless-stopped
    pull_policy: always
    environment:
      REDIS_URL: demo-cache
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
      RABBITMQ_HOST: demo-queue
      POKE_API_BASE_URL: https://pokeapi.co/api/v2
      COLLECTOR_ENDPOINT: http://otel-collector:4317
      NPM_RUN_COMMAND: rpc
    ports:
      - 8082:8082
    healthcheck:
      test: ["CMD", "lsof", "-i", "8082"]
      interval: 1s
      timeout: 3s
      retries: 60
    depends_on:
      postgres:
        condition: service_healthy
      demo-cache:
        condition: service_healthy
      demo-queue:
        condition: service_healthy
  # Demo End
```

## Tracetest

The `docker-compose.yaml` includes two services related to Tracetest.

```yaml
version: "3"
services:
  # Cloud-based Managed Tracetest
  tracetest-agent:
    image: kubeshop/tracetest-agent:latest
    environment:
      # Get the required information here: https://app.tracetest.io/retrieve-token
      - TRACETEST_API_KEY=${TRACETEST_TOKEN}
      - TRACETEST_ENVIRONMENT_ID=${TRACETEST_ENVIRONMENT_ID}
  tracetest-apply:
    build:
      dockerfile: Dockerfile.tracetest
    volumes:
      - ./resources:/resources
    environment:
      TRACETEST_TOKEN: ${TRACETEST_TOKEN}
      TRACETEST_ENVIRONMENT_ID: ${TRACETEST_ENVIRONMENT_ID}
    entrypoint:
      - bash
      - /resources/apply.sh
    networks:
      default: null
    depends_on:
      demo-api:
        condition: service_started
      tracetest-agent:
        condition: service_started

  tracetest-run:
    build:
      dockerfile: Dockerfile.tracetest
    volumes:
      - ./resources:/resources
    environment:
      TRACETEST_TOKEN: ${TRACETEST_TOKEN}
      TRACETEST_ENVIRONMENT_ID: ${TRACETEST_ENVIRONMENT_ID}
    entrypoint:
      - bash
      - /resources/run.sh
    networks:
      default: null
    depends_on:
      tracetest-apply:
        condition: service_completed_successfully

  # Tracetest End
```

The `collector.config.yaml` file contains the OpenTelemetry Collector configuration that enables routing traces from the Pokeshop API to Tempo.

```yaml
# collector.config.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
    timeout: 100ms

exporters:
  logging:
    loglevel: debug
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]
```

How do traces reach Grafana Tempo?

The Pokeshop API code uses the native Node.js OpenTelemetry modules which send information to the OpenTelemetry Collector to be processed and then sent to Grafana Tempo.

## Grafana Tempo

The `docker-compose.yaml` includes 3 services related to Grafana Tempo.

- OpenTelemetry Collector
- Grafana Tempo
- Grafana (Dashboards)

```yaml
version: "3"

# ...
services:

  # ...
  # Grafana
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.59.0
    command:
      - "--config"
      - "/otel-local-config.yaml"
    volumes:
      - ./collector.config.yaml:/otel-local-config.yaml
    depends_on:
      - tempo
  tempo:
    image: grafana/tempo:latest
    command: [ "-config.file=/etc/tempo.yaml" ]
    volumes:
      - ./tempo.config.yaml:/etc/tempo.yaml
      - ./tempo-data:/tmp/tempo
    ports:
      - "3200:3200"   # tempo http
      - "9095:9095"   # tempo grpc
      - "4317"  # otlp grpc
      - "4318"  # otlp http
  grafana:
    image: grafana/grafana:9.4.3
    volumes:
      - ./grafana.config.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
      - GF_FEATURE_TOGGLES_ENABLE=traceqlEditor
    ports:
      - "3000:3000"
  # Grafana End
```

## Run the Pokeshop API, Grafana and Tracetest

To start all the services, and run the Tracetest tests, run this command:

```bash
docker compose run tracetest-run
```

Make sure to use the `http://demo-api:8081/` URL in your test creation, because your Pokeshop API and Tracetest are in the same network.

## Trace-Based Tests

Check out the `resources/test.yaml` file.

```yaml
# resources/test.yaml

type: Test
spec:
  id: ZVJwkpC4g
  name: Pokeshop - Import
  description: Import a Pokemon
  trigger:
    type: http
    httpRequest:
      method: POST
      url: http://demo-api:8081/pokemon/import
      body: '{"id":6}'
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[tracetest.span.type="http"]
    name: "All HTTP Spans: Status  code is 200"
    assertions:
    - attr:http.status_code = 200
  - selector: span[tracetest.span.type="general" name="import pokemon"]
    name: Validate that this span always exists after the message queue
    assertions:
    - attr:tracetest.selected_spans.count  =  1
    - attr:span.events not-contains "exception"
  - selector: span[tracetest.span.type="database" name="get pokemon_6" db.system="redis" db.operation="get" db.redis.database_index="0"]
    name: Validate that Redis is using Charizard.
    assertions:
    - attr:db.payload = '{"key":"pokemon_6"}'
  - selector: span[tracetest.span.type="database" name="create postgres.pokemon" db.system="postgres" db.name="postgres" db.user="postgres" db.operation="create" db.sql.table="pokemon"]
    name: Validate that the Postgres has Charizard.
    assertions:
    - attr:db.result contains "charizard"
```

## View Trace Spans Over Time in Grafana

To access a historical overview of all the trace spans the Pokeshop App generates, jump over to Grafana on `http://localhost:3000`.

With Grafana and Tracetest you can run trace-based tests and automate running E2E and integration tests against real trace data. And, use Grafana to get a historical overview of all traces your distributed application generates.

## Learn More

Feel free to check out our [examples in GitHub](https://github.com/kubeshop/tracetest/tree/main/examples) and join our [Slack Community](https://dub.sh/tracetest-community) for more info!
